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TITLE OF THE INVENTION 
MARKER LAYOUT METHOD, MIXED REALITY APPARATUS, AND 
MIXED REALITY SPACE IMAGE GENERATION METHOD 



5 FILED OF THE INVENTION 

The present invention relates to a method of 
laying out a marker used as a position index upon 
presenting a mixed reality space and, more particularly, 
to a marker layout method which allows easy detection 
;;L':; 10 of a marker even when a plurality of players share a 

^==1' mixed reality space, and a mixed reality apparatus 

;0 using the same, 

'^0: BACKGROUND OF THE INVENTION 

;;q; 15 In recent years. Mixed Reality (MR) that aims at 

seamlessly coupling a real space and virtual space has 
been extensively studied. MR has received a lot of 
attention as a technique which aims at realizing 
coexistence of a virtual reality (VR) world that can be 
20 expressed in only a situation isolated from a real 
space, and the real space, and augments VR. 

A typical apparatus for implementing MR is a 
head-mounted display (HMD) . More specifically, MR is 
implemented by mixing and displaying real and virtual 
25 spaces on the HMD. MR schemes using the HMD include an 
optical see-through scheme for superposing a CG image 
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or the like onto an image displayed on a 
semi-transparent ( see- through) HMD, and a video 
see- through scheme for mixing a CG image or the like 
onto image data sensed by a video camera attached to an 
5 HMD, and then displaying the mixed image on the HMD. 

MR can be used, quite possibly, in new fields 
which are qualitatively quite different from VR, such 
as a medical assist application for presenting the 
state inside the body of a patient to a doctor as if it 

10 were seen through, a job assist application for 

superposing and displaying the assembly sequence of a 
product on real parts in a factory, and the like, 

A technique for removing "deviation" between real 
and virtual spaces is commonly required for these 

15 applications. "Deviation" can be classified into 
positional deviation, temporal deviation, and 
qualitative deviation, and many studies have been 
conventionally made for removal of positional deviation 
as the most fundamental requirement among those 

20 deviations. 

Especially, in case of video see-through MR, 
since an image process scheme can be relatively easily 
applied as a method of correcting positional deviation, 
alignment using an image process has been 

25 conventionally proposed. 
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More specifically, a method in which markers 
which are marked in color so an image process can 
easily detect are laid out at a predetermined position 
in a real space, and the viewpoint position is computed 
5 on the basis of the marker positions detected from an 
image sensed by a camera attached to a player, and a 
method of correcting an output signal from a 
location/posture sensor based on marker positions in an 
2 image, and the like are available. 

1: 10 When markers in image data are detected, and the 

si 

'\ location/posture of a player is estimated based on the 

Q detection result, the markers must appear in an image 

to have an appropriate size and nearly uniform 
y intervals. Also, since a sufficient number of markers 

n 15 must be simultaneously detected in the image upon 

% computing the viewpoint position, the markers must be 

laid out to be observed in the image at somewhat narrow 

intervals , 

On the other hand, in order to improve the 
20 tracking or identification precision of markers, the 

markers must be laid out to be observed in the image at 
somewhat broad intervals. 

In case of a single player, it is not so 
difficult to lay out markers to satisfy the 
25 aforementioned conditions. However, in an application 
which allows a plurality of players to share a common 
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MR space, markers which are laid out at equal intervals 
at positions that can be observed from a given player 
cannot often be observed at equal intervals from 
another player . 
5 For this reason, Japanese Patent Laid-Open 

No, 11-84307 has proposed an arrangement in which in an 
air hockey game system in which two players hit and 
return a virtual puck toward each other on a table as a 
real object, markers having different colors in units 

10 of players are provided to allow the individual players 
to observe the markers with a preferred layout and size. 

However, marker layout using different colors 
becomes harder with increasing number of players who 
share a single MR space. More specifically, in order 

15 to detect a given color by an image process, the colors 
of the markers and background object, and those of 
markers in units of users must be easily detected and 
extracted by the image process. However, when the 
number of colors used increases, it becomes difficult 

20 to satisfy such conditions, and extraction errors and 
identification errors among markers can occur. 

Fig, 10 is a graph for explaining recognition 
error factors when a plurality of different color 
markers are used. In Fig. 10, the abscissa plots red, 

25 and the ordinate plots green. For the sake of 

simplicity, a blue axis is not shown. In Fig. 10, 
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region A defines the color distribution of marker type 
A (red marker) , and region B defines the color 
distribution of marker type B (orange marker) . In this 
manner, when the number of players increases and 
5 similar colors are used as markers, even though a 
player observes a red marker, the marker may be 
detected as color of region B (i.e., of orange marker), 
thus causing detection errors. 

When a plurality of markers having different 
10 colors in units of players are laid out, a very large 
number of markers appear in the real space, and may 
complicate the vision of the player, thus impairing 
reality upon experiencing an MR space. 



15 SUMMARY OF THE INVENTION 

It is an object of the present invention to 
provide a marker layout method in a mixed reality space, 
which can solve the aforementioned problems, and can 
reliably detect markers in units of players even when a 

20 plurality of players share a common mixed reality space. 

It is another object of the present invention to 
provide a marker layout method in a mixed reality space, 
which can present the number of markers from simply 
increasing even when the number of players increases . 

25 It is still another object of the present 

invention to provide a mixed reality apparatus that 
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exploits the marker layout method in the mixed reality 
space according to the present invention. 

It is still another object of the present 
invention to provide a mixed reality apparatus and a 
5 mixed reality space image generation method, which can 
solve the aforementioned problems, and allow a player 
to experience mixed reality without being bothered by 
the presence of markers. 

More specifically, a gist of the present 

10 invention lies in a marker layout method for laying out 
markers in a real space as position indices upon 
generating a mixed reality space, comprising the step 
of: laying out the markers to have a positional 
relationship that allows a given player not to observe 

15 markers to be used by only another player when a 

plurality of players who observe the mixed reality 
space within different movable ranges observe the mixed 
reality space. 

Another gist of the present invention lies in a 

20 mixed reality apparatus for computing and/or correcting 
location/posture information of a player using markers 
laid out by a marker layout method of the present 
invention. 

Still another gist of the present invention lies 
25 in a storage medium that stores a marker layout method 
in a mixed reality space according to the present 
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invention as a program which can be executed by a 
computer . 

Still another gist of the present invention lies 
in a mixed reality apparatus for making a player 
5 experience mixed reality by making the player observe a 
mixed reality space image obtained by mixing real and 
virtual spaces, markers serving as position indices 
being laid out in the real space, the apparatus 
J comprising: marker detection means for detecting the 

;J 10 markers from image data obtained by sensing the real 

space from a substantially viewpoint position of the 
Q- player; and mixed reality space image generation means 

for generating the mixed reality space image to be 
y: observed by the player, so the player observes virtual 

g; 15 object images that do not include any images of the 

I markers in surrounding regions (marker regions) 

including the markers in the image data. 

Still another gist of the present invention lies 
in a mixed reality space image generation method for 
20 generating a mixed reality space image which makes a 

player experience mixed reality by mixing a real space 
in which markers serving as position indices are laid 
out, and a virtual space, comprising: the marker 
detection step of detecting the markers from image data 
25 obtained by sensing the real space from a substantially 
viewpoint position of the player; and the mixed reality 
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space image generation step of generating the mixed 
reality space image to be observed by the player, so 
the player observes virtual object images that do not 
include any images of the markers in surrounding 
5 regions (marker regions) including the markers in the 
image data. 

Still another gist of the present invention lies 
in a storage medium which stores a program that can be 
executed by a computer, and makes the computer that 

10 executes the program function as a mixed reality 
apparatus of the present invention. 

Still another gist of the present invention lies 
in a storage medium that stores a mixed reality space 
image generation method according to the present 

15 invention as a program which can be executed by a 
computer . 

Other features and advantages of the present 
invention will be apparent from the following 
description taken in conjunction with the accompanying 
20 drawings, in which like reference characters designate 

the same or similar parts throughout the figures thereof. 



BRIEF DESCRIPTION OF THE DRAWINGS 
The accompanying drawings, which are incorporated 
25 in and constitute a part of the specification, 

illustrate embodiments of the invention and, together 
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with the description, serve to explain the principles 
of the invention* 

Fig. 1 is a block diagram showing an example of 
the arrangement of an MR apparatus to which a marker 
5 layout method according to the present invention can be 
applied; 

Fig. 2 is a view for explaining the types and 
locations of devices the player wears; 

Figs. 3A and 3B are views for explaining a game 
10 in an embodiment of the present invention; 

Fig. 4 is a view for explaining a marker layout 
method according to the present invention; 

Figs. 5A to 5C are views showing markers observed 
from the individual players in the layout shown in 
15 Fig. 4; 

Figs. 6A to 6C are views showing markers observed 
from the individual players when markers are laid out 
in the same layout as in Fig. 4 without using any 
obstacles; 

20 Fig. 7 is a graph showing the color region of a 

red marker in the embodiment of the present invention; 
Fig. 8 is a flow chart showing the processing 

sequence for detecting a marker in the color region 

shown in Fig. 7 from color image data; 
25 Figs. 9 A to 9C are views for explaining a marker 

deletion method; and 
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Fig, 10 is a graph for explaining detection 
errors when the number of colors of markers is 
increased. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
A preferred embodiment of a marker layout method 
according to the present invention will be described 
hereinafter with reference to the accompanying drawings. 
In this embodiment, the present invention is applied to 
an MR game played by three players while sharing a 
single virtual space, but a marker layout method in an 
MR space according to the present invention can be 
applied to any other arbitrary applications. 

Fig. 1 shows the arrangement of an MR apparatus 
for implementing an MR game to which a marker layout 
method of the present invention is applied. Fig, 1 
shows the arrangement for three players. The MR 
apparatus comprises player processors 100 to 300 
provided in units of players, and a controller 400 
connected to the player processors. The number of 
player processors connected to the controller 400 
increases with increasing number of players. 

The player processors 100 to 300 have an 
identical arrangement. That is, each player processor 
comprises I/O blocks {102, 103, 105, 1011, 1021, 1022) 
such as sensors, display device, and the like attached 
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to the player's body, and circuit blocks (101, 104, 106, 
107) for processing signals captured by the sensors, 
and generating an image to be displayed on the display 
device on the basis of the signal processing results 
and information of other players supplied from the 
controller . 

The arrangement of the player processor will be 
explained below with reference to Fig. 1 and Fig. 2 
that shows the types and locations of I/O devices each 
player wears in this embodiment . 

As shown in Fig. 2, each player wears on his or 
her head a head-mounted image display device (to be 
referred to as an HMD hereinafter) 105 as a display 
device for mixing and displaying real and virtual 
spaces. In this embodiment, the HMD can be either a 
video or optical see- through type. In the following 
description, a case using the video see-through HMD 
will be explained. 

Two small video cameras 103 are provided to those 
portions of the HMD 105, that are near player's eyes. 
Video data sensed by these video cameras at nearly the 
same viewpoint as that of the player are supplied to an 
image mixing unit 106 (to be described later) via an 
image input unit 104, and are displayed on the HMD 105 
of the player after being superposed on a virtual space 
image. The image input unit 104 also supplies the 



input images to a head location detecting (measuring) 
unit 101 in addition to the image mixing unit 106. 

A head location/posture sensor 1011 comprising, 
e.g., a magnetic sensor, is also attached to the head 
5 of the player. The head location/posture sensor 1011 
can be attached using, e.g., the HMD 105. The output 
signal from the head location/posture sensor 1011 is 
input to the head location/posture measuring unit 101. 
The head location/posture measuring unit 101 detects 
10 position correction markers using the image supplied 

from the image input unit 104, and corrects the signal 
supplied from the head location/posture sensor 1011, 
thus estimating the viewpoint ' position and posture of 
the player . 

15 On the other hand, an interactive input device 

102 is attached to a player's arm. The interactive 
input device 102 has a location/posture sensor 1021 for 
detecting the location and posture of a portion where 
the device 102 is attached, and a switch (trigger) 1022 

2 0 which is turned on/off in response to the motion of the 
attached portion and is used by the player to input 
commands by making predetermined actions. 

In the following description, a case will be 
exemplified below wherein a plurality of players (three 

25 players in this embodiment) share an identical MR space, 
play a game for defeating enemies while ducking away 
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from attacks of enemies appearing in the MR space and 
competing with other players for the number of enemies 
he or she shoots down or score until the time or damage 
by enemy attacks reaches a given level, and can input 
5 the following commands using the interactive input 
device 102 , 

•Command 1 (Sight Command) 

A command for displaying a line of sight 

indicating the sight position in the virtual space. 
10 This command is input by locating the wrist above the 

elbow with the back of the hand facing up, 

•Command 2 (Shooting Command) 

A command for shooting the sight position 

indicated by the line of sight. This command is input 
15 by reciprocally moving the arm (from the elbow to the 

palm) back and forth at a given acceleration or higher 

while the line of sight is displayed by the sight 

command. 

•Command 3 (Defense Command) 
20 A command for defending against an enemy's attack. 

This command is input by directing the fingertips 

upward with the back of the hand facing the enemy. 

•Command 4 (Reset Command) 

A command for re- inputting the sight command 
25 after the shooting command is input. This command is 

input by having the arm hanging down. 
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More specifically, as routine actions in the game 
of this embodiment, the player repeats command inputs 
in a cycle of the sight command shooting command 
reset command — > sight command, and inputs the defense 
5 command as needed in this cycle. 

The command input by the interactive input device 
102 is supplied to an image generation unit 107. 

The image generation unit 107 transfers the head 
location/posture information of the player supplied 

10 from the head location/posture measuring unit 101 and 

command information supplied from the interactive input 
device 102 to the controller 400. Also, the unit 107 
generates a virtual space image to be displayed on the 
HMD 105 of the corresponding player using the head 

15 location/posture information and command information of 
that player, and the head location/posture information, 
command information, and model information of other 
players, the location, moving direction, and state 
information of enemy characters, and information of the 

20 locations, shapes, and the like of obstacles laid out 
in the space, which are received from the controller 
400, and outputs the generated image to the image 
mixing unit 106. 

The image mixing unit 106 mixes the image (real 

25 space image) captured by the video cameras 103 attached 
at the positions near the view point of the player, and 
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the virtual space image generated by tine image 
generation unit 107, and supplies the mixed image to 
the HMD 105 of the player. 

The controller 400 comprises an MR space managing 
5 unit 1 to which the aforementioned player processors 
100 to 3 00 are connected, and a real object location 
measuring unit 2 . The MR space managing unit 1 
distributes information that pertains to the head 
locations and postures of the players, and the 

10 locations, postures, and commands of the interactive 
input devices 102, which are received from the player 
processors 10 0 to 3 00, and executes game processes such 
as appearance/disappearance control of enemy characters 
to be displayed in the virtual space, scoring of 

15 shooting input by the shooting command, and the like. 
Information that pertains to the models, locations, 
moving directions, and states (e.g., defeated or not) 
of enemy characters are distributed to all the users 
connected together with each player information. 

20 When real objects 31 to 3 3 are laid out as 

obstacles to shooting to adjust the game difficulty 
level, as shown in Figs. 3A and 3B, the MR space 
managing unit 1 also manages information that pertains 
to the shapes and locations of these real objects 31 to 

25 33. 
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Fig. 3A is a perspective view of a game field (MR 
space) viewed from a certain player, and Fig. 3B is a 
top view of the game field. Figs, 3A and 3B show a 
case wherein three real space objects 31 to 33 are laid 
5 out as obstacles on a table 3 0 (in the real space) . In 
this embodiment, since the sight command is input by- 
raising the wrist above the elbow using the interactive 
input device 102, as described above, the real space 
objects are laid out on the table 3 0 which is as high 
10 as the waist level of a standard body. However, the 
^-'l need for the table can be obviated depending on command 

K input actions using the interactive input device. 

In Figs. 3 A and 3B, the real space objects 31 and 

C J- 

fy 32 are fixed in position, and the object 33 is movable. 

rg; 15 If a movable real space object is provided, the game 

Q. 

difficulty level can be dynamically changed. For 
example, when the movable real object 33 moves to the 
right and left at a random speed, it is more difficult 
for a player to shoot an enemy character as a target 

20 than a case wherein only the still real objects 31 and 
32 are laid out. The movement of the movable real 
object may be controlled by either the MR space 
managing unit 1 or another control circuit. In this 
case, the model of the movable object is managed by the 

25 MR space managing unit 1, and its location is obtained 
by measuring an object location sensor 331 provided to 
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that movable object 33 by the real object location 
measuring unit 2 , 

The aforementioned MR apparatus can be 
implemented by a client-server system which includes 
5 the controller 400 as a server, and (the circuit 
sections of) the player processors 100 to 300 as 
clients* Since processes that pertain to each player 
are distributed to and executed by each client, the 
apparatus can flexibly cope with an increase/decrease 

10 in the number of players. More specifically, the 
player processor can be implemented by a versatile 
computer having a video I/O function and a signal 
reception function from various sensors, and the 
controller 400 can be implemented by a versatile 

15 computer having an interface that can communicate with 
each player processor, and a measurement signal 
reception function from the object location measuring 
unit 2 . 

However, since computations pertaining to 
20 three-dimensional image display must be done in real 

time, a relatively fast computer having an accelerator 
dedicated to such computations (so-called 3D 
accelerator) or the like is preferably used. Also, the 
controller 400 and player processors 100 to 300 are 
25 preferably connected via communication lines having a 

large capacity such as 100BASE-T. If the communication 
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line has a small capacity, the processing speed drops 
larger with increasing number of players. 
(Marker Layout Method) 

Fig. 4 is a perspective view showing an example 
5 of a marker layout in this embodiment. In the game of 
this embodiment, real objects serving as obstacles are 
laid out. By laying out markers using these obstacles, 
the aforementioned conditions can be satisfied while 
^=3- restricting markers which enter the field of view of 

10 each player within the moving range of the player, 
Cn^ Figs. 5A to 5C show extracted markers players A, 

B, and C can respectively see in Fig. 4. In this 
manner, since each player can observe markers at nearly 
fij, equal intervals and in a quantity required for 

k-:^ 15 computing the viewpoint position, but cannot see 

% markers for other players, the markers need not use 

different colors. Also, as shown in Fig. 4, a 
plurality of players can share identical markers. 

Figs . 6A to 6C show markers observed by the 
20 individual players in correspondence with Figs. 5A to 
5C when no obstacles are used. As can be clearly 
understood from comparison between Figs. 5A to 5C and 
Figs. 6A to 6C, the marker layout method according to 
the present invention can make the number of markers 
2 5 each player observes very small, and satisfy the 
aforementioned conditions. 
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An increase in the number of players can be coped 
with by changing the shapes (sectional shapes, heights, 
and the like) of real objects to which the markers are 
provided, or adding another color. Even when the 
5 number of colors increases, since one color is not 
assigned per player, markers for a large number of 
players can be laid out using fewer colors . 

The marker layout positions may be determined 
manually but can be determined by generating real 

10 object models and viewpoint position movable range 
models of the individual players in advance, and 
obtaining a range in which a given player can see but 
the lines of sight of other players are intercepted. 
Furthermore, the marker layout positions may be 

15 computed using the number of markers to be provided and 
layout rules. Conversely, obstacle shapes and/or 
layout positions that satisfy the aforementioned 
conditions may be computed. 

Even when no objects such as obstacles that can 

20 be used to lay out markers are available, real objects 
may be laid out at positions where no problem is posed 
in terms of an application, and are hidden by ^virtual 
space images, whereby the player can experience an MR 
space without being bothered by the presence of real 

25 objects for markers. Hiding of markers and the like 
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using virtual space images will be explained in detail 
later. 

(Detection of Marker) 

A marker detection method will be explained below. 
5 Fig. 8 is a flow chart showing the flow of processing 
for detecting markers of type A (red markers) having 
color included in region A shown in Fig. 7. The head 
location/posture measuring unit 101 executes a marker 
detection process and uses the detected marker 
10 information. 

Images sensed by the video cameras 103 are 
captured via the image input unit 104 (step S7 01) . The 
captured images are then binarized (step S702) , More 
specifically, let 1 be pixels included in region A 
15 shown in Fig. 7 (a blue axis is not shown) , and 0 be 
pixels included in the remaining region. That is, if 
li: i-th pixel that forms input color image I 
Ri, Gi, Bi: R, G, and B values that form li 
ITHi : i~th pixel value of binary image 
20 RminA, GminA, BminA: R, G, and B minimum values 

that define region A 

RmaxA, GmaxA, BmaxA: R, G, and B maximum values 
that define region A 

Then, a binary image ITH is formed by setting 1 in ITHi 
25 corresponding to li that satisfies RminA < Ri < RmaxA, 
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GminA < Gi < GmaxA, and BminA < Bi < BmaxA, and 0 in 
ITHi corresponding to another li in units of li , 

The binary image ITH undergoes a labeling process 
to extract marker regions (clusters) (step S703) . A 
5 center of gravity (Xn, Yn) and area an of each cluster 
are computed (step S704) , and are output to a viewpoint 
position/posture estimation module (not shown) in the 
head location/posture measuring unit 101 (step S705) , 
The viewpoint position/posture estimation module 

10 corrects the output signal from the head 

location/posture sensor 1011 on the basis of absolute 
coordinate positions of markers, which are registered 
in advance, and the marker positions (centers of 
gravity of clusters) detected from the image, thus 

15 computing the viewpoint position and posture of the 
player , 

In Fig, 8, a process upon detecting markers in a 
given color (red) has been explained. When the number 
of players increases and there are a plurality of 
20 different color markers, detection processes in units 
of colors are repeated by changing a threshold value 
upon generating a binary image* 
(Deletion of Marker) 

As described above, markers are not necessary if 
25 the output from the head location/posture sensor 1011 
is accurate. Furthermore, the presence of markers in 
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an MR space is preferably not recognized since sense of 
reality may suffer if the player recognizes markers. 

For this reason, the present invention is 
characterized by apparently deleting markers to make 
5 the player unconscious of the presence of markers. As 
a method of apparently deleting markers, various 
methods may be used. Of these methods, a method of 
superposing a virtual image on each marker and 
presenting the virtual image to the player is 

10 preferable since it can minimize the processing load 
and unnatural feeling. 

Figs. 9A to 9C are views for explaining the 
marker deletion method. Fig. 9A shows markers for 
player A, which are laid out according to the marker 

15 layout method in this embodiment, and corresponds to 
Fig. 5A. In order to delete markers laid out on real 
objects such as a table, obstacles, and the like, a 
prospective layout place is sensed using a video camera, 
digital camera, still camera, or the like before 

20 markers are laid out so as to capture image data as a 
texture, and the markers are then laid out. The MR 
space managing unit 1 manages, as virtual objects for 
hiding the markers, two-dimensional patches each of 
which has a position, direction, and size to hide each 

25 marker, and a texture image corresponding to the marker 
before marker layout, and distributes them to the 
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player processors 100 to 300. The image generation 
unit 107 in each play processor renders such patches as 
parts of a virtual space image using the head 
location/posture information of the player upon 
5 rendering the virtual space image pertaining to the 
game. As a result, the image mixing unit 10 6 
superposes images before marker layout on regions 
corresponding to the markers in the real space image 
(Fig. 9B) , 

10 In this manner, by replacing/superposing image 

data, the player can concentrate on the game without 
being bothered by the presence of the markers. Image 
data prepared as texture data are preferably captured 
under conditions (light source position and brightness) 

15 actually used to reduce unnatural feeling of vision. 

But texture images for deleting markers to be laid out 
on real objects made of an identical material may use a 
single common texture. Texture images are not 
particularly limited as long as they do not make the 

20 player feel unnatural. Also, texture images to be laid 
out for the game may be used to hide the markers, and 
three-dimensional objects that hide the markers may be 
laid out as virtual objects in place of two-dimensional 
patches , 

25 In an application that does not use any obstacles, 

when real objects are laid out for only the purpose of 
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marker layout, or when it is hard to superpose virtual 
objects on individual markers, a virtual object 91 that 
covers all real objects may be used, as shown in 
Fig. 9C. In such case, a virtual object to be used can 
5 be appropriately determined depending on applications. 
For example, in a game application, a stage or 
something like that in a virtual space may be formed 
using the virtual object 91. Or in an application in 
± which the user is preferably unaware of any difference 

H- 10 from a real space, object data sensed in the real space 

J 

fr may be used. 

y 

g [Other Embodiments] 

In the above embodiment, information obtained 
H: from the markers is used to correct errors of the head 

5 15 location/posture sensor 1011, Alternatively, the 

^ viewpoint position and posture of the player may be 

computed from only the information obtained from the 
markers without using the head location/posture sensor 
1011, and the present invention can be used in such 
20 application . 

The above embodiment has exemplified a video 
see- through MR apparatus, but the present invention can 
be similarly applied to an optical see-through MR 
apparatus . 

25 In the above embodiment, single-color markers are 

used, but any other visible features may be used. For 
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example, a plurality of different colors may be used, 
or features other than color may be used. Also, 
specific textures, shapes, or the like may be used, and 
these features may be combined. 

The objects of the present invention are also 
achieved by supplying a storage medium (or recording 
medium) , which records a program code of a software 
program that can implement the functions of the 
above-mentioned embodiments to a system or apparatus, 
and reading out and executing the program code stored 
in the storage medium by a computer (or a CPU or MPU) 
of the system or apparatus. In this case, the program 
code itself read out from the storage medium implements 
the functions of the above-mentioned embodiments, and 
the storage medium which stores the program code 
constitutes the present invention. The functions of 
the above-mentioned embodiments may be implemented not 
only by executing the readout program code by the 
computer but also by some or all of actual processing 
operations executed by an OS (operating system) running 
on the computer on the basis of an instruction of the 
program code. 

Furthermore, the functions of the above-mentioned 
embodiments may be implemented by some or all of actual 
processing operations executed by a CPU or the like 
arranged in a function extension board or a function 



extension unit, which is inserted in or connected to 
the computer, after the program code read out from the 
storage medium is written in a memory of the extension 
board or unit. 

To restate, according to the present invention, 
markers which are used in an alignment process in an MR 
apparatus can be laid out while satisfying required 
positional and numerical conditions, even when a 
plurality of players share a single MR space. 

Since the marker color need not be changed for 
each player, probability of occurrence of detection 
errors can be reduced even when the number of players 
increases . 

Furthermore, since real space objects which are 
laid out originally can be used, no extra objects used 
to lay out markers are required depending on 
applications . 

As many apparently widely different embodiments 
of the present invention can be made without departing 
from the spirit and scope thereof, it is to be 
understood that the invention is not limited to the 
specific embodiments thereof except as defined in the 
appended claims. 



WHAT IS/CLAIMED IS: 

1 ' / A marker layout method for laying out markers in 
ayreal space as position indices upon presenting a 
4tiixed reality space, comprising the step of: 
5 laying out the markers to have a positional 

relationship that allows a given player not to observe 
markers to be used by only another player when a 
plurality of players who observe the mixed reality 
space within different movable ranges observe the mixed 

10 reality space, 

2, The method according to claim 1, wherein a 
visible feature of the markers to be used by only the 
other player is similar to a visible feature of the 
markers used by the given player. 

15 3. The method according to claim 2, wherein the 

visible feature includes at least one of color, texture, 
shape, and size of the marker, 

4. The method according to claim 1, wherein the 
markers to be used by only the given player are laid 

20 out at positions hidden by real objects when the 
markers are observed from the other player, 

5, The method according to claim 4, wherein the real 
objects are laid out for an application that uses the 
mixed reality space. 
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6. The method according to claim 1, wherein the 
markers to be laid out include markers shared by a 
plurality of players, 

7. The method according to claim 1, wherein the 
5 markers have a common color. 

8. A mixed reality apparatus for computing and/or 
correcting location/posture information of a player 
using markers laid out by a marker layout method of 
claim 1. 

10 ^ A computer readable storage medium which stores a 
^marker layout program for laying out markers in a real 
space as position indices upon presenting a mixed 
reality space, storing: 

a program code of laying out the markers to have 
15 a positional relationship that allows a given player 
not to observe markers to be used by only another 
player when a plurality of players who observe the 
mixed reality space within different movable ranges 
observe the mixed reality space, 
20 1(/. A mixed reality apparatus for making a player 

/experience mixed reality by making the player observe a 
mixed reality space image obtained by mixing real and 
virtual spaces, 

markers serving as position indices being laid 
2 5 out in the real space, 

said apparatus comprising: 
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marker detection means for detecting the markers 
from image data obtained by sensing the real space from 
a substantially viewpoint position of the player; and 
mixed reality space image generation means for 
5 generating the mixed reality space image to be observed 
by the player, so the player observes virtual object 
images that do not include any images of the markers in 
surrounding regions (marker regions) including markers 
G in the image data, 

in^ 10 11. The apparatus according to claim 10, wherein said 

CT mixed reality space image generation means substitutes 

m overlays images of the marker regions by 

predetermined virtual object images. 

12, The apparatus according to claim 11, wherein said 
15 mixed reality space image generation means substitutes 

or overlays an image of a region including all the 
detected markers by a predetermined virtual object 
image . 

13. The apparatus according to claim 11, wherein the 
20 predetermined virtual object images are plane patches 

with the same or similar texture, size, location and 
posture of the marker regions before the markers are 
laid out . 




A mixed reality space image generation method for 



25 generating a mixed reality space image which makes a 

player experience mixed reality by mixing a real space 
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in which markers serving as position indices are laid 
out, and a virtual space, comprising: 

the marker detection step of detecting the 
markers from image data obtained by sensing the real 
5 space from a substantially viewpoint position of the 
player; and 

the mixed reality space image generation step of 
generating the mixed reality space image to be observed 
by the player, so the player observes virtual object 
10 images that do not include any images of the markers in 
surrounding regions {marker regions) including the 
markers in the image data. 

15. The method according to claim 14, wherein the 
mixed reality space image generation step includes the 

15 step of substituting or overlaying images of the marker 
regions by predetermined virtual object images. 

16. The method according to claim 14, wherein the 
mixed reality space image generation step includes the 
step of substituting or overlaying an image of a region 

20 including all the markers in the image data by a 
predetermined virtual object image. 

17. The method according to claim 15, wherein the 
predetermined virtual objects are images are plane 
patches with the same or similar texture, size, 

25 location and posture of the marker regions before the 
markers are laid out . 
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1^. A computer readable storage medium that stores a 
/program which can be executed by a computer, making the 
computer which executes the program function as : 

a mixed reality apparatus for making a player 
5 experience mixed reality by making the player observe a 
mixed reality space image obtained by mixing a real 
space in which markers serving as position indices are 
laid out, and a virtual space, comprising: 

marker detection means for detecting the markers 
10 from image data obtained by sensing the real space from 
a substantially viewpoint position of the player; and 

mixed reality space image generation means for 
generating the mixed reality space image to be observed 
by the player, so the player observes virtual object 
15 images that do not include any images of the markers in 
surrounding regions (marker regions) including the 
markers in the image data. 

* ^ computer readable storage medium which stores a 
mixed reality space image generation program for 
20 generating a mixed reality space image which makes a 

player experience mixed reality by mixing a real space 
in which markers serving as position indices are laid 
out, and a virtual space, storing: 

a marker detection program step of detecting the 
25 markers from image data obtained by sensing the real 
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space from a substantially viewpoint position of the 
player; and 

a mixed reality space image generation program 
step of generating the mixed reality space image to be 
5 observed by the player, so the player observes virtual 
object images that do not include any images of the 
markers in surrounding regions (marker regions) 
including the markers in the image data. 

D1- 
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ABSTRACT OF THE DISCLOSURE 
There is disclosed a marker layout method in a 
mixed reality space, which can reliably detect markers 
in units of players, even when a plurality of players 
5 share a common mixed reality space. According to this 
invention, markers to be used by only a given player 
are laid out at positions that cannot be seen from 
other players. Real objects used in an application 
that uses a mixed reality space may be used. 
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